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Abstract 

The mirror game has been recently proposed as a simple, yet powerful 
paradigm for studying interpersonal interactions. It has been suggested that 
a virtual partner able to play the game with human subjects can be an ef¬ 
fective tool to affect the underlying neural processes needed to establish the 
necessary connections between the players, and also to provide new clinical 
interventions for the rehabilitation of patients suffering from social disor¬ 
ders. Inspired by the motor processes of the central nervous system (CNS) 
and the musculoskeletal system in the human body, in this paper we develop 
a novel interactive cognitive architecture based on nonlinear control theory 
to drive a virtual player (VP) to play the mirror game with a human player 
(HP) in different configurations. Specifically, we consider two cases: the 
former where the VP acts as leader and the latter where it acts as follower. 
The crucial problem is to design a feedback control architecture capable of 
imitating and following or leading a human player (HP) in a joint action task. 
Movement of the end-effector of the VP is modeled by means of a feedback 
controlled Haken-Kelso-Bunz (HKB) oscillator, which is coupled with the 
observed motion of the HP measured in real time. To this aim, two types of 
control algorithms (adaptive control and optimal control) are used and imple¬ 
mented on the HKB model so that the VP can generate human-like motion 
while satisfying certain kinematic constraints. A proof of convergence of 
the control algorithms is presented in the paper together with an extensive 
numerical and experimental validation of their effectiveness. A comparison 
with other existing designs is also discussed, showing the flexibility and the 
advantages of our control-based approach. 
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1 Introduction 


The emergence of coordinated behavior between humans is a common phenomenon 
in many areas of human endeavor. Examples include improvisation theater, group 
dance, music playing, team sports and parade marching |[I1. At the core of the in¬ 
teraction between the players lies a fundamental feedback mechanism where each 
player adapts his/her motion in response to the observed movement of the other. 

To study this intriguing phenomenon, the mirror game has been recently pro¬ 
posed as a simple, yet effective paradigm. In its simplest formulation, the mirror 
game features two people imitating each other’s movements at high temporal and 
spatial resolution |!2|. The game can be played in different experimental condi¬ 
tions: the former where one of the players leads and the other has to follow the 
leader movement (Leader-Follower condition); the latter where the two players 
create joint synchronized movement (Joint Improvisation condition). 

The theory of similarity in social psychology suggests that people prefer to 
team up with others possessing similar morphological and behavioral features, and 
that interpersonal coordination is enhanced if their movement shares similar kine¬ 
matic features 1311 . Further evidence suggests that motor processes caused by 
interpersonal coordination are strictly related to mental connectedness. To be spe¬ 
cific, motor coordination between two people contributes to social attachment Q. 

As suggested in f5|, coordination games can therefore be used to help people 
suffering from social disorders to improve their social skills. Also they can be 
effectively exploited in social robotics to enhance attachment, coordination and 
rehabilitation during human-robot interactions |^. For this reason, it has been 
proposed that creating a VP or avatar able to coordinate its motion with that of a 
HP can be extremely useful to study the onset of coordination and how it is affected 
by similarity/dissimilarity between the players’ motion characteristics ["Jl. A VP 
can also be used for diagnostics and rehabilitation of patients suffering from social 
disorders as recently proposed in [8|. 

The aim of this paper is the design of a novel interactive cognitive architecture 
(ICA) based on nonlinear control theory able to drive a VP to play the mirror game 
with a human either as a leader or as a follower. Specifically, the goal is that of 
designing a cognitive architecture able to drive the motion of the VP interacting 
with a HP in real-time while exhibiting certain desired kinematic features. When 
playing as a follower, the ICA needs to guarantee that, while exhibiting the desired 
movement properties, the VP tracks as closely as possible the motion of the hu¬ 
man leader. When playing as the leader, the ICA needs instead to generate new 
interesting motion. In both cases, it is crucial for the VP to engage with the HP 
by producing human-like response in terms of kinematics (maximum acceleration, 
velocity profile etc) and delay times. In this paper we take the view that the de¬ 
sign of such an architecture is fundamentally a nonlinear control design problem 
where given some reference input the architecture has to drive the VP onto a de¬ 
sired motion which is a function of the movement of the human player being sensed 
during the game. In particular, the ICA can be integrated into the humanoid robot 
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to achieve the desired dual-arm coordination 

We explore two different approaches, one based on adaptive control, the other 
on optimal control. Our control architecture mimics the two fundamental actions 
which have been suggested to be at the core of the emergence of motor coordination 
between two or more effectors in biological systems: feedback and feedforward 
lfT0] - fT2l . Specifically, the motor system is able to correct the deviation from the 
desired movement with the aid of feedback control, whilst feedforward control 
allows it to reconcile the interdependency of the involved effectors and preplan the 
response to the sensory incoming information ifTTUTSll . 

It is shown experimentally that the proposed control architectures are able to 
effectively drive the VP to play the mirror game while generating motion with 
desired kinematic properties. In particular, we use the concept of Individual Motor 
Signature (IMS) recently proposed in |[T4l[T5]l to characterize the motion of an 
individual player and evaluate how similar/dissimilar the motion of two different 
individuals is. Following our approach we are able to show that the VP driven by 
the cognitive architecture presented in the rest of this paper can play the mirror 
game either as a leader or a follower while exhibiting a desired IMS. 

Relevant previous work in the literature includes the generation of human-like 
movement llThll . the development of a mathematical model to explain the coordi¬ 
nation dynamics observed experimentally in the mirror game f2l, and the Human 
Dynamic Clamp paradigm proposed in |[7lll71ll81l where the use of a virtual partner 
driven by appropriate mathematical models is proposed to study human motor co¬ 
ordination. These previous approaches will be used to investigate and compare the 
performance of the novel strategy presented in this paper. We wish to emphasize 
that the control algorithms developed and validated in what follows can be also ef¬ 
fectively used for trajectory planning to enhance human-robot coordination in joint 
interactive tasks. 

The rest of the paper is organized as follows. The mirror game set-up, problem 
statement and motor signature are discussed in Section [2] before presenting the 
schematic of the proposed cognitive architecture in Section [3] The feedback control 
strategies at the core of the ICA are developed and analyzed in Section |4] and |5j 
The experimental validation of the control algorithms is presented in Section 
where experimental results are discussed showing the effectiveness of the proposed 
strategies. A comparison with other existing approaches is also carried out. Finally, 
conclusions and suggestions for future work are drawn in Section |7] 

2 The Mirror Game Problem 

Investigation of interpersonal coordination requires appropriate experimental paradigms. 
A typical paradigm recently proposed in the literature is the mirror game, which 
involves two people imitating each other’s movements at high temporal and spa¬ 
tial resolution f2l- It can be played in two different conditions: Leader-Follower 
condition, where the follower attempts at tracking the leader motion as accurately 
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Figure 1 Experimental set-up of the mirror game between a VP and a HP at the 
University of Montpellier, France (see f8] for further details). 


as possible, and Joint Improvisation condition, where the players jointly coordi¬ 
nate and synchronize their movements without any of the two being designated as 
leader or follower. 

Our set-up is inspired by the one in |21|. Specifically, a small orange ball is 
mounted onto a string, which the HP can move back and forth along the string 
itself. In the meanwhile, the VP on the opposite screen moves its own ball on a 
parallel string with the same length (see Fig. [T]l- In this implementation of the 
mirror game, two players (a HP and a VP) are required to move their respective 
ball back and forth and synchronize their movement. Here, we assume that the 
game is played in a Feader-Follower condition, where the HP is the leader and the 
VP (robot or computer avatar) is the follower trying to track the leader movement. 
However, the VP can opt to act as the leader as well. 

The position of the ball moved by the HP is detected by a camera. A feedback 
control strategy then needs to be designed in order to generate the trajectory of the 
ball moved by the VP so as to track the movement of the ball controlled by the HP. 
Such a trajectory can then be provided to the on-board controllers of the VP (robot 
or computer avatar) as the desired trajectory for its end effector. 

To solve this control problem so that the VP motion presents similar features 
to the motion of a human player, we need to choose an appropriate model of the 
VP motion that can then be controlled using a nonlinear feedback strategy. To this 
purpose, here we use the Haken-Kelso-Bunz oscillator, which was first proposed 
in |[T9]| as a model able to capture the observations made in experiments on human 
bimanual coordination. The model consists of two nonlinearly coupled nonlinear 
oscillators described by 

z -h {at^ -h - y)z + co^z =[a + b{z - w)^] (z -w) (1) 

where z,z represent the position and velocity of finger 1, w,w the position and ve¬ 
locity of finger 2 (modeled by a replica of the equation above obtained by swapping 
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w,vv with z,z); a and b are the coupling parameters and a, j 8 , 7 and co character¬ 
ize the response of each uncoupled finger when subject to some reference signal. 
However, it is worth pointing out that, other than describing intrapersonal motor 
coordination, the HKB model has been also used to describe interpersonal motor 
coordination involving two different people ll20ll^ . In particular, the HKB os¬ 
cillator has been suggested in the literature as a paradigmatic example of human 
motor coordination ||7][19l. Solving the mirror game can then be formulated as the 
following control problem. Given a nonlinear HKB oscillator of the form 

y =-(ay^ + -Y)y-(O^x + u 

where x and x refer to the position and velocity of the end effector of the VP, 
respectively, and u is an external control input, the problem is to design a feedback 
controller u such that x{t) achieves bounded asymptotic tracking of the position of 
the HP, while expressing some desired kinematic features. 

As metrics to characterize the kinematic properties of the motion of an indi¬ 
vidual playing the game we use the concept of individual motor signature (IMS), 
recently introduced in II14II15L It has been shown that the IMS is time invariant and 
is unique for each player. It is defined in terms of the velocity profile (or disfribu- 
fion) of fhe player’s mofion in solo frials. To quantify how similar or dissimilar fhe 
signafures of fwo differenf players are, we use the earth mover’s distance (EMD) 
between any two probability distribution functions (PDF) of their velocity time se¬ 
ries II151I22L The EMD between two PDFs pi and p 2 can be computed as follows 

EMD{puP 2 ) = ^ \CDFpPz)-CDFp,{z)\dz 

where Z denotes the integration domain, and CDFp. (z) denotes the cumulative dis¬ 
tribution function of the distribution p, , / € {1,2}. Fig|2}a) shows the position time 
series of the same HP and the corresponding PDF of velocity in two solo trials. It 
is visible that the two PDFs of velocity time series resemble each other in terms 
of their shape, and the EMD between them is 0.024. In contrast, the two PDFs of 
velocity time series in Figl2}b) differ remarkably from each other, and the value of 
EMD is 0.604, which confirms fhe qualification of fhe PDF of velocify fime series 
in solo frials as individual motor signafure. 

3 Design of the Cognitive Architecture 

We design fhe cognitive architecture of the VP so as to replicate the main processes 
involved in making a human being play the mirror game (see Fig. O. The visual 
system detects the ball’s position on the string and generates visual signals, which 
are then transmitted to the central nervous system (CNS including brain and spinal 
cord). Several parts of the CNS (such as ventral horn, cerebellum and motor cortex) 
use an internal model to predict the kinematic characteristics of the other player’s 
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Figure 2 Position time series and the corresponding PDF of velocity time series 
for the same player (a) and two different players (b). 
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Figure 3 Motor coordination between two players in mirror game. 
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motion and generate the neural impulses that control the extension and contrac¬ 
tion of muscles. Finally, the neuromuscular system activates and coordinates the 
muscles involved in generating the hand movements. 

This architecture is mapped onto the real-time control schematics shown in Fig. 
|4] whose blocks are briefly described below. 


A camera is used to detect the position of the HP, say 

A filtering and velocity estimation block is used to filter the position data 
acquired by the camera via a low pass filter and to estimate the velocity of 
the HP (reference) via the simple formula 


. rpitk) - rp{tk-i) _ ^ 

rv{t) = - - - t G [tk,tk+l] 


(3) 


where k ^ N*, and T = tk — tk-\ denotes the sampling period of the camera. 
The estimated velocity is then used to predict the HP position over the next 
interval by using the expression: 


rp{t) =rp{tk) + rv{t){t-tk), t G [tkfik+i] 


(4) 


As an alternative, we could adopt a nonlinear observer to provide a better 
prediction of the reference velocity; for example, the nonlinear extended ob¬ 
server in 1231. Here we find that such a complication is unnecessary to solve 
the problem of interest and therefore choose to use the simple yet effective 
estimation strategy discussed above. 

• At the core of the architecture lie the two blocks Temporal Correspondence 
Control and Signature Control. The former is designed to regulate the end 
effector model so that its motion tracks that of the HP with varying degrees 
of dynamic similarity. Specifically, it aims at minimizing the position er¬ 
ror between the time series of the HP and that of the VP. The latter block 
uses the prerecorded velocity time series of a reference HP with the desired 
IMS (velocity profile) in order to generate the avatar trajectory with desired 
kinematic features. In particular, the aim of the signature controller is that 
of reducing the distance (computed in terms of EMD) between the velocity 
distribution of the VP and that of some reference HP it aims at replicating 
the motion characteristics of. 

• The prerecorded velocity trajectory of a reference HP playing solo repre¬ 
senting the desired IMS is stored in the Signature generator block while the 
signature of the avatar motion is estimated by the Signature estimation block. 

• The end effector model is used to generate the avatar motion via an appro¬ 
priate feedback control scheme. As mentioned before, we use the HKB os¬ 
cillator to describe the dynamics of the end effector. 
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Figure 4 Block diagram of the cognitive architecture of VP. Green blocks allow 
for the control of temporal correspondence between the VP and the HP, and blue 
blocks take into account the desired individual motor signature. 


• Finally, the output of the cognitive architecture (position and velocity .r and 
x) is used as the reference motion for the VP. 

In what follows we focus on the design of the feedback control strategies that 
drive the cognitive architecture. We derive and compare two different types of 
controllers. First we develop an adaptive algorithm able to control the tempo¬ 
ral correspondence between the VP and the HP during the game (green blocks in 
Fig®- Then, we consider an optimal controller to solve simultaneously the multi 
objective control problem of tracking the trajectory of the HP while preserving the 
features of the desired IMS of interest (both green and blue blocks in Fig®. For 
both strategies a proof of convergence is given before presenting numerical and 
experimental investigation of their performance. 

4 Adaptive Control of Temporal Correspondence 

To solve the control problem of temporal correspondence, we propose an adaptive 
controller based on the end effector model shown in ([2]). Specifically, we choose 
the nonlinear controller given by 

u= [a{t)+b{t){x-rp)^]{x-ry)-Cpe^^''^~'''’^^{x-rp) (5) 

Coordination Temporal Correspondence 

where Vp is the position of the HP, fy is the estimated velocity, Cp and 5 are constant 
parameters while the coupling parameters a and b are updated according to the 












































Table 1 Adaptive Control Algorithm 


1 : setk = \ and running time 
2: while (time < Ts) 

3: detect the position ^^( 4 ) of HP 

4: estimate the velocity 4 ( 4 ) of HP with ([3]) 

5: generate the control signal u with (l5]l 

6 : update coupling parameters a and b with ® and O 

7: obtain the position ;c and velocity x of VP by solving (O 

8 : k = k+ I 

9: end while 


adaptive laws: 


d = -e ^‘‘[(x-rp)(y-ry) + riaix-rpf]-ria (6) 

and 

b = [co^x + {ay^ + ^x^ - y)y - T]^(y - 4 ) - u] - t]^ (7) 

where r]a is a positive constant. Note that the control law (|5l) consists of two com¬ 
plementary terms. The first has the same structure as the one of the coupling pro¬ 
posed in lfT9l to model the interaction between two HPs, albeit with the introduc¬ 
tion of adaptive parameters to account for variability between different HPs. The 
second term, depending on the fixed paramefers Cp and 5, deals wifh fhe posifion 
error when fhe velocify mismafch approaches zero and hence fhe firsl ferm decays 
fo zero. When |i: — fy | is relafively large, fhe coupling ferm of fhe HKB equation 
insfead dominafes and mofor coordinafion befween fhe fwo players becomes more 
pronounced during fhe mirror game. 

Theorefical analysis of fhe adaptive confrol algorifhm in Table [T] is given in 
whaf follows below. 

4.1 Convergence analysis 

Theorem 4.1. The adaptive feedback controller (|5l) ensures that the solution of the 
controlled HKB model (|2]) satisfies 


Ht)-rp{t)\ t€[0,+oo) 


if rja is chosen so that 
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where 


E{^) = \ [(^(0) - + (3^(0) - ^v(0))2 + 


and 


e=^up{T^ + l){r,{kT)-r,{kT-T)Y. 
keW 


Proof: Choose the energy-like function 


E = 


A 1 


2 L 


{x-Vpf + {y-rvY + -f 


( 8 ) 


Note that ky is fixed in each sampling interval [kT, (k-|- l)r),k € N*. Then the time 
derivative of E along the trajectories of ^ with u defined in (|5l) is given by 

E = {x - rp){x - ry) + {y - ry)y + e^“d + e^^b 

= {x-rp){y-ry)-{y-ry)[{ay^+^x^ - /jy + (O^x - u] 

+ (y - by) [(O^x + (ay^ + - 7)y -rjaiy- by) -u]- rj^e^^ 

- [(x-rp)(y-ry) + T]a(x-rp)Y -rjae^^ 

= -ria(x - rp)^ -Vaiy- by)^ - r]ae^‘' - Vae^'’ 

= -2r]aE, tG[kr,(k+l)r) 


Solving the above differential equation yields 

E{t)=e-^^-^'-’^^^E{kT), f € [kr,(k+l)r) (9) 


Moreover, at the sampling point kT we have 
E{kT)-E-{kT) 

= \[{x- rp{kT))^ -{x- rp{kT -T)- by{kT - T)T)^ 

+ {y-ry{kT))'^-{y-by{kT-T))^] 

< {rp{kT) - VpikT -T)- by{kT - T)T)'^ 

+ {ry{kT) - by{kT - T))^+E-{kT) 

= (1 + T^ibyikT) - by{kT - T))^+E-{kT) 

which is equivalent to 

E{kT) <e + 2E-{kT) (10) 

where 

£= sup{T^ + l){by{kT)-by{kT-T))^ 
keN* 

and 

E-(kT) = lim E(t) 

t\kT 
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Evaluating ® and (fTOl) ait = {k+\)T and nesting the inequalities backwards till 
i = 0 , we get 


E{kT) < e 


1 + 






1 - 


= e- 




1 - 


2 

e-^aT 


+ 



E(0) 


<- 

“ 1 - 


£ 

r~ 

girjaT 


+ 




E(0), 




( 11 ) 


when the inequality rja > ^ holds. Moreover, combining (|9ll with (fTTI) . we get 

E(0<- -+ ie[0,+oo) 

which clearly implies 




2 e 


+ ■ 


g2rj„7’ _ 2 grjaT 


^vm, 


t e [0,+oo) 


Remark 4.1. It is easy to demonstrate that the coupling parameters a and b are 
upper bounded with the proposed adaptive laws. 

Remark 4.2. Since rp{t) € [0,/],Vt > 0 and |rv(0l — 7 > the following inequality 
holds 

£ = sup (r2 + l){r,{kT) - f,{kT - T)f < ^ 

jteN* '' 

where I refers to the length of the string. Generally, the upper bound for the posi¬ 
tion error is relatively conservative. When the velocity of the HP is small, £ is small 
as well, and the estimation for the position error is accurate enough. In addition, 
taking the limit of (II 111 as kT oo and combining it with ([8]), the position error 
between the two players satisfies the following inequality: 

limsup \x{t) — rp{t) \ < 
kT —)-+oo 

Similarly, we can estimate the velocity error as 

m -K{,)\<eV,J ^ ,/m, >e[ 0 .+~) 

While being effective in achieving bounded tracking of the HP position, the 
control approach derived so far is unable to explicitly guarantee that the generated 
motion follows a desired IMS. Therefore we consider a different scheme based on 
optimal control. 


l£ 


g2r\aT _ 2 
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5 Optimal Temporal Correspondence and Signature Con¬ 
trol 

The second approach we propose is based on optimal control, which is an effective 
framework to allow for movement coordination and reconcile target tracking and 
individual motor signature 1241. We assume that in the mirror game the motion of 
the VP can be decomposed into a series of goal-directed movements (see Fig. |5]l in¬ 
fluenced by both the position of the HP and the desired individual motor signature. 
Thus, we formulate the problem of driving the end effector motion as described by 
(IH) on a finite time interval [t/tTtr+i] as the dynamic optimization problem 

min/ (12) 

uGR 

where 

J = ^ {xitk+\) - rp{tk+-i))^ 

^ '' -V-^ 

Temporal Correspondence /io\ 

1 f^k+\ ^ ^ 

+ o (4 (t) - rcs{z)Y +rimu{TYdz 

Similarity 

with the constraint dp + Q(y = \ and 6p, 0^, > 0 being tunable control parame¬ 

ters. Here, rp{tk+i ) denotes the estimated position of the HP at time tt+i (see ([H) for 
further details), while refers to a prerecorded velocity time series representing 
the desired motor signature. For the sake of simplicity, the optimization interval 
[tk,tk+i] in the cost function (fTSl) corresponds to the sampling interval. The above 
cost function mainly consists of three terms. The first term aims at minimizing 
the mismatch between the position time series of the HP and that of the VP. The 
second term takes care of making the velocity profile of the VP as close as possi¬ 
ble to the reference one (motor signature). The last term guarantees boundedness 
of the control effort. In particular, the idea behind this cost function is that the 
human-like movement of the VP emerges from the integration of three different 
goals related to temporal correspondence, motor signature and control energy ex¬ 
penditure, respectively. Notice that the VP acts as a leader when dp is close to 0, 
since the term related to the position error x — tp in the cost function is negligible 
and the only aim of the VP is to exhibit the motion characterized by the desired 
motor signature. Instead, the VP behaves as a follower if 6p is close to 1 as in this 
case the controller aims solely at minimizing the mismatch between the HP and 
VP terminal positions. 

5.1 Convergence Analysis 

To prove stability of the optimal control algorithm in Table |2j we focus on proving 
the boundedness of the position error between the reference input and the output of 
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Figure 5 Movement of the VP end effector in the mirror game 
Table 2 Optimal Control Algorithm 


1 : set A: = 1 and running time 
2: while (time < Ts) 

3: detect the position rp( 4 ) of HP 

4: estimate the position rp{tk+i) of HP with ([Hi 

5: generate the control signal u by solving (fT^ 

6 : obtain the position x and velocity x of VP by solving ^ 

7: k = k+\ 

8 : end while 


the cognitive architecture. In particular, optimality of the cost function is guaran¬ 
teed in each optimization interval if the damped harmonic oscillator is adopted as 
end effector model instead of the HKB oscillator. Since both the reference position 
Vp and the desired velocity are bounded, we assume Vp G [r, f] and € [y, v]. 

Theorem 5.1. The optimal control algorithm applied to the HKB oscillator (O 
with cost function diJI) ensures bounded position error between the HP and the VP. 


Proof: First of all, we need to demonstrate that there exists a limit cycle in the 
HKB oscillator 

f x = y 

\ y = — {ax^ + — Y)y — aPx 

Choose the energy-like function as follows 


V{x,y) 


Gp-x^ + y^ 
2 


The time derivative of V{x,y) along the trajectory of the HKB oscillator (fldh is 
given by 


V(x,y) = oPxx + yy 

= oPxy - {ax^ -I- Py^ - y)y^ - oPxy 
= -{ax^ + py^-y)y^ 
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Define 


f~majc ■ 


:= max 



) ^min ■ 


:= min 



and construct a region tH as follows (see Figure O 

:= {(xjj) G : Cl < V{x,y) < C 2 } 


where the positive constants ci and C 2 satisfy 


f~min 


= max 



Clearly, contains no stationary points of the system. Indeed, the only stationary 
point of the system is (x,y) = ( 0 , 0 ), but this stationary point is located outside 
of the region Moreover, V{x,y) > 0 when V{x,y) = ci and V{x,y) < 0 when 
V{x,y) = C 2 - According to the Poincare-Bendixson theorem, we can conclude that 
the HKB oscillator (fldl) has a limit cycle in 01. 

Let J* denote the value of the cost function (fTSl) with the optimal control algo¬ 
rithm in each time interval, and let Jq represent the value of the corresponding cost 
function when u = 0. Since u aims at minimizing the value of the cost function for 
all ^ € N*, we can write 


r < Jo = 



Recall that Xp is bounded, which indicates that is bounded according to 

equation Q and (|4ll. Moreover, ^^(t) is bounded as well, and note that x{t) and 
x{t) are also bounded since the trajectory of the HKB oscillator converges to the 
limit cycle in 91. Thus, we can claim that Jq is bounded for k G N*, which implies 
boundedness of J* and as a consequence of the position error between the VP and 
the HP. ■ 

Remark 5.1, It is demonstrated that the bound on the tracking error \x{tk+i) — 
Xp{tk+i) \ converges to 0 as dp ^ \ and ?],„ —)■ 0. Similarly, the bound of the velocity 
error |i:(4+i) — rc{tk+\)\ goes to 0 j/ 0(j —^ h dm ^ 0 and r^fk) = y{tk) (see 
Supplementary Material/or the detailed analysis). 

The analytical solution for the optimization problem (fT^ is available if a linear 
damped harmonic oscillator is employed as the end effector model, and Pontrya- 
gin’s minimum principle provides necessary and sufficient conditions to solve the 
minimization problem |[25l . 

Corollary 5.1. Given the linear system 


x + ax + bx = u 


the optimal control approach guarantees convergence to the optimum solution over 
each subinterval. 
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Figure 6 Illustration on the construction of region IH. The black ellipse is de¬ 
scribed by the equation + j3y^ = 7 , and the region iR refers to the ring-shaped 
area bounded by two red ellipses corresponding to V{x,y) = c\ and V{x,y) = C 2 , 
respectively. 


Proof: According to the fundamental theorem of the calculus of variations, 
we need to examine the second variation of the given cost function in order to 
establish the optimum. From the conclusions in ESll . the second variation of the 
cost function (fT3l) is given by 


6^J= Gp[5x(tk+i)f 
+ ( 5X 5u ) 


Hxx Hxu 
Hlx Huu 


5X 

5u 


dt 


where H is the Hamiltonian 


H{X,u,X) 




+ -r]u^ + X^ 


y ) 

—ay — bx + u J 


with X = [x,xY = [x,yY and A = [^ 1 ,^ 2 ]^. Rewrite the linear system in matrix 
form as follows 

X=AX+Bu 


where 



Let X =X* + 5X and u = u* + 5u, where X* and u* denote optimal state and 
optimal control, respectively. Since X* =AX* +Bu*, we get 

5X=A5X + B5u (15) 
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where 5X = [8x^8xY. Thus, it follows from Hxu = H^x = [0 0]^, Huu = T] > 0 
and 

"- = (o 

that 

= dp[8x{tk+i)f + r^' 8X{tYHxx8X{t) + r]{8u{t)fdt 

^tk 

rh+i 

= dp[8x{tt+i)f + / eY8x{t)Y + ri{8u{t)Ydt 
Jik 

>0 

Moreover, 8^J = 0 is equivalent to 5x(4+i) = 0, 8x{t) = 0 and 8u{t) = 0 for all 
t € [tkdk+i], which yields 8x{t) = 8x{tk) = 0 from (fTSl) . This corresponds to the 
optimal solution X* and the optimal control u*. Therefore, we conclude that the 
optimal control ensures the minimum value of the cost function (fT^ for the linear 
system in each time interval. ■ 

6 Validation 

In this section we experimentally validate our control algorithms on a simple, yet 
effective, set-up implemented at the University of Bristol, UK. Experimental data 
of human-human interaction is used to evaluate the matching performance of the 
VP and a comparison with existing VP models is provided as well. 

6.1 Experimental Set-up 

The employed set-up was developed for measuring motions of players in the one¬ 
dimensional mirror game. A human participant is required to join the game while 
interacting with a VP (implemented on a laptop computer). In order to detect the 
position of his/her hand, a leap motion controller ETll is employed (see Fig. |7]). The 
leap motion controller and the laptop computer are both placed on a table whose 
height is around 70cm. The HP is required to wave his/her hand horizontally over 
the leap motion controller at a vertical distance of approximately 50cm. Indeed, at 
this distance the horizontal resolution of the device is maximum and it is able to 
measure the hand position within a range of 60cm. The position of the hand of the 
HP within this interval is mapped intto the interval [—0.5,0.5] and visualized on 
the computer screen as a green solid circle, while the position of the VP is visual¬ 
ized as a blue solid circle. The adaptive control algorithm described in Section |4] 
is implemented with Euler method in Matlab (version R2012b), whilst the solver 
“bvp4c” is adopted to handle the optimization problem of the optimal control al¬ 
gorithm presented in Section [5l Players can be either standing or seated. After the 
game is initialized, there is a 2s wait before data recording begins and the game 
starts. This initial delay is used to allow the HP to place his/her hand over the leap 
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Leap motion 
controller 


Figure 7 Experimental set-up of the mirror game between a HP and a VP. The 
position of the human fingertip rp{t) is deteeted by a leap motion controller, and 
the sampled position rp{kT) is sent to the computer, while the position x{t) of the 
VP is generated by implementing the control algorithm. Two balls are shown on 
the computer screen, which describe the end effector positions of the HP (green 
ball) and the VP (blue ball), respectively. 

motion controller. Human players are not instructed before playing the game, but 
they are just told to act as a leader and let the VP follow them during a 60s round. 
The case where the VP acts as a leader is also tested experimentally. 

6.2 Measures 

The temporal correspondence between the VP and the HP is evaluated according 
to the following indexes: the root mean square (RMS), the relative position error 
(RPE) lITSl . the circular variance (CV) |[28]l and the time lag (TE) |[29l . 

1. RMS: The root mean square of the position error between two players de¬ 
scribes the tracking accuracy of the follower in the mirror game. 

RMS= ^-X2,iy 

where n is the number of sampling steps in the simulation, and xij and X 2 ,i 
denote the positions of the leader and the follower at the /-th sampling step, 
respectively. 

2. RPE: The relative position error is a measure of how well the follower was 
tracking the leader in the mirror game ifTSl . Positive values of the RPE indi¬ 
cate that the follower is indeed behind the leader. 



{xi{t) -X 2 {t))sgn{xi{t)), 

ifsgn{xi{t)) =sgn{x 2 {t)) ^0; 
\xi{t) —X 2 (t) I, otherwise. 


where xi{t) and X 2 {t) {x\{t) and X 2 {t)) are the positions (velocities) of the 
leader and follower at time t, respectively. 
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3. CV: The circular variance is used to quantify the coordination level between 
two players 


CV = 





€[ 0 , 1 ] 


where represents the relative phase between two players at the ^-th sam¬ 
pling step, n refers to the total number of time steps and || • || denotes the 
2 -norm. 


4. TL: The time lag describes the shifted time that achieves the maximum cross¬ 
covariance of the two time series. It is sensitive to the changes of motion 
direction and hence can be interpreted as the average reaction time of the 
player in the mirror game |[29l . 


6.3 Results 


6.3.1 VP driven by the ICA based on adaptive control 

The parameters for the HKB equation and the adaptive feedback controller (AFC) 
in (IDl are set heuristically as follows: a = 10, j3 = 20, 7 = —1, ft) = 0.1, a(0) = —5, 
b{0) = —5, Cp = 40 and 5 = 0.25. In our implementation the sampling time is 
T = 0.1s and therefore = 30 > ^ ~ 0.35. In particular, the values of all the 
previous parameters have been chosen so that the response of the HKB oscillator 
to several sinusoidal signals with different frequencies is qualitatively the same as 
the one of a HP trying to track the same references. Moreover, it is worth pointing 
out that the initial values of a and b influence the performance of the avatar only at 
start-up. 

The reactive-predictive controller (RPC) proposed in O is also implemented 
to compare its performance against that of our adaptive feedback controller when 
considering the same input trajectory from the human leader. Following the scheme 
presented in El to implement the RPC, the dynamics of the VP is described by the 
following system: 

5 

X = ^Ai(OiCos{(Oit) + f 
i=l 

where x € M represents the position of the avatar and 

f = k{ry—x), k>0 


with the parameters A,- being estimated adaptively as 

5 


Ai = X 


fy — ^Aisin{(Oit) 


sin{(Oit), 


A >0 


;=i 

As suggested in [2], in this case the parameters are chosen as follows: COi = 0.025, 
o >2 = 0.05, ft )3 =0.075, (O 4 = 0.1, (O 5 = 0.125, A =0.01, k = 30 and A,(0) = 0,V/ = 

1,...,5. 
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(a) position time series (b) relative phase time series 

Figure 8 Time series of the position (a) and of the relative phase (b) between the 
human leader and the VP; blue (AFC), red (RPC), green (human leader) 



(a) veocity PDF 


Figure 9 Distributions of the velocity 
human leader and the VP; blue (AFC), 



(b) relative phase PDF 


and the relative phase (b) between the 
(RPC), green (human leader) 
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(a) Temporal Correspondence 


(b) Similarity 


Figure 10 Time evolution of positions and relative phase (a) and PDF of velocities 
(b) while the VP is driven by the optimal control and acts as follower in the mirror 
game 


To compare the performance of the two algorithms, we plot the time series of 
both the position and the relative phase (see Fig. [8ll together with the distributions 
of velocity and relative phase of the human and the VP (see Fig. |2l- In particular, 
the relative phase between the two players is defined as A<p = ^hp — <}>vp, where 
^HP and (/Vp are the phases of the HP and the VP, respectively. In addition, the 
phase is estimated according to the method proposed in 1^ . Note that positive 
values of A0 correspond to the avatar following the HP during the game. 

We can observe that, when using the reactive-predictive controller, the position 
of the VP presents oscillations away from the human participant position not only 
when he/she is moving, but also when he/she is still. Such an oscillatory feature 
does not appear when using the adaptive feedback controller. In general, both the 
position error e = x — rp and the velocity error e = x — r^ turn out to be higher when 
using the RPC strategy. When using the adaptive feedback controller, the position 
error remains smaller never exceeding 0.2, while it can become as high as 0.8 when 
using RPC. Similarly, the velocity error never exceeds 0.62 for the AFC, while it 
goes up to a maximum of 3 for the RPC. 

Moreover, when using AFC, the relative phase time series is much closer to 0 
than that obtained when using RPC, meaning that with our proposed algorithm it 
is possible for the VP to better synchronize with the human leader. Such results 
are confirmed by fhe relative phase disfribufions obfained when using bofh fhe al- 
gorifhms, as shown in Fig. |3b). Finally, fhe difference in fhe velocify disfribufions 
of fhe HP and fhe VP is much more evidenf when RPC is used, confirming fhaf 
our sfrafegy beffer capfures fhe feafures of fhe HP and is fherefore able fo replicafe 
more accurafely fhe kinemafic properties observed in human motor coordination in 
fhe confexf of fhe mirror game. 
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(a) Temporal Correspondence 


(b) Similarity 


Figure 11 Time evolution of positions and relative phase (a) and PDF of velocities 
(b) while the VP is driven by the optimal control and acts as leader in the mirror 
game 


6.3.2 VP driven by the ICA based on optimal control 

The parameters of the VP are set heuristically as follows: a = l,j8 = l,7=l, 
(0 = 1, rjin = 10^^ and the sampling period T = tk+\ — 4 = 0.03s. In order for the 
VP to play the mirror game as a follower, we set the control parameters dp = 0.9 
and da = 0.1, which makes the VP play in a follower configuration (minimizing 
the position mismatch more than the signature EMDs). As we can see from the 
top panel in Fig fTOl al. the VP performs quite well as a follower during the game; 
indeed, the root mean square (RMS) of the position error is equal to 0.057. In 
order to distinguish the leader from the follower in the game, we also calculate the 
relative phase between the HP and the VP. From the bottom panel in Fig. fTOf a) 
we can observe that the majority of relative phase is positive, meaning that the VP 
is following the HP in the game for most of the time. The circular variance (CV) 
is also calculated to take into account the coordination level between two players. 
The CV between the HP and the VP is 0.95, which indicates a high coordination 
level. As for the distribution of the velocity, we can see in Fig. [TOl' b) that the VP 
signature (blue line) is closer to that of the HP (red line) than the desired motor 
signature (cyan line). This is due to the choice of the control parameters in the cost 
function (fTSl) that render the strategy able to minimize more the position error be¬ 
tween the players. The measured EMDs at the end of the trial are given as follows: 
EMD(S/g,//P) = EMD(S/g, VP) = 0.017 and EMD(VP,7/P) = 0.005. 

The VP can be enabled to play the game as a leader by changing the control 
parameters setting = 0.1 and da = 0.9. Experimental results are shown in Eig. 
M The RMS of the tracking error is 0.08, and the CV between the two players 
is 0.81. As depicted in the bottom panel of Eig. [TIT a), the majority of the relative 
phase time series is negative, meaning that now the VP is leading the HP during 
the game for most of the time. In contrast to the previous case, the velocity distri¬ 
butions shown in Pig. fTlT bi confirm that the VP is now matching well the desired 
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Figure 12 Position time series of the HP pair (upper panel) and the corresponding 
VP pair (lower panel) in the mirror game. 


signature (velocity profile). In this case the trade off is slightly larger but still the 
relevant HMDs are given as follows: EMD{Sig,HP) =EMD(5'/g, VP) = 0.004 and 
EMD(VP,//P) = 0.008. 

6.3.3 Interaction between two VPs 

As mentioned before, our final goal is fo create a customized VP able fo “replicate” 
fhe kinematic fealures of a given HP in fhe mirror game. To fesf how well fhe VP 
can replicafe fhe fealures of a given HP, we carried oul fhe following experimenl. 
Eirsl of all, Iwo HPs are required fo play fhe mirror game in a Eeader-Eollower con¬ 
dition. Then fhe signafures of fhe human leader (HE) and fhe human follower (HE) 
are fed into a virfual leader (VE) and a virfual follower (VE), respectively. Einally, 
we make fhe VE {6p = 0.43) and fhe VE {6p = 0.92) play fhe mirror game fogelher. 
In Eig. [121 the upper panel shows fhe time evolution of fhe posifion frajecfories of 
fhe HE and fhe HE, while fhe lower panel presenfs fhe posifion frajecfories of fhe 
corresponding VE and VE. If appears lhal fhe VE and fhe VE succeed in mafching 
fhe kinemalic characlerisfics of fhe HE and fhe HE in lerms of fhe RMS value of 
fheir position error [RMS{HL,HF) = 0.0466 and RMS{VL,VF) = 0.0497] and 
fhe time lag befween fhe fwo players [TL{HL,HF) = 0.09 and TE(VL,VP) = 
0.09]. In addition. Table [3] gives fhe values of EMDs and describes fhe mafching 
resulls quanlifafively. Sigi and Sigf represenf fhe signafures of HE and HE when 
playing solo, respectively. If shows lhal fhe proposed approach allows to replicate 
effectively fhe game dynamics befween fwo humans playing fhe mirror game via 
fwo coupled VPs. 
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Table 3 Matching performance of VPs in terms of EMD. 


EMD(5/gL,HE) 

0.010 

EMD(5/gf,HP) 

0.007 

EMD(5/gL,VE) 

0.006 

EMD(5/gf,VP) 

0.006 

EMD(HE,HP) 

0.0034 

EMD(VE,HE) 

0.0052 

EMD(VE,VP) 

0.0031 

EMD(VP,HP) 

0.0053 


Table 4 Indexes of temporal correspondence 



HP 

OPC 

HKB 

RPC 

JKE 

RPE 

0.0914 

0.0816 

2.1767 

0.3838 

0.1467 

CV 

0.3011 

0.1002 

0.9400 

0.7602 

0.4859 

TE 

0.2035 

0.1274 

1.5192 

0.0428 

-0.9674 


6.3.4 Comparison with existing models 

In order to compare the VP models, we need to establish a benchmark, which 
describes the general kinematic characteristics of human participants in the mir¬ 
ror game. To this aim, 5 human participants were asked to track a prerecorded 
reference signal, and indexes of temporal correspondence were recorded to repre¬ 
sent a benchmark of typical human dynamics. The existing VP models were then 
enabled to track the same reference. Corresponding indexes were computed for 
the VPs and compared with the benchmark to evaluate the proposed control algo¬ 
rithms. We adopted the following models to drive the VP: our strategy based on 
optimal control (OPC), Haken-Kelso-Bunz model (HKB) ifTTl . reactive-predictive 
control (RPC) [21 and Jirsa-Kelso excitator (JKE) [71. The parameter setting of 
existing VP models is the same as that in the relevant references. In Table |4j we 
show the benchmark of temporal correspondence and performance indexes of the 
corresponding VP models. On the whole, our algorithm performs best in terms 
of matching the benchmark among all the VP models. In addition to temporal 
correspondence, we also computed how similar the VP signature is to that of the 
HP during the benchmark experiment (see PigfTT]). We find again that the opti¬ 
mal control strategy developed in this paper is the best in terms of replicating the 
human-like movement with an estimated EMD(//P, OPC) = 0.0184. 

7 Conclusions 

We presented the novel design of an interactive cognitive architecture able to drive 
a virtual player to play the mirror game against a human player. Two strategies 
were developed. The first, based on adaptive control, was shown to be effective 
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Figure 13 PDF of velocity time series for different VP models. 


to achieve temporal correspondence between the motion of the virtual player and 
that of the human individual. Convergence of the algorithm was proved. It was 
noticed that the adaptive control strategy does not allow the VP to exhibit some 
desired kinematic features (individual motor signature) of a given human player. 
To overcome this limitation, a different strategy based on the iterative solution of 
an appropriate optimal control problem was proposed. After proving boundedness 
and convergence of this additional approach, its effectiveness was tested experi¬ 
mentally. It was shown that the proposed strategy is able to drive the VP so as to 
play the game both as leader or follower while matching well the individual motor 
signature of a given individual. Finally, a comparison with other existing models 
was carried out confirming the effectiveness of the proposed approach. We wish to 
emphasize that our approach opens the possibility of making VPs, each modeling 
a different individual, play against each other and produce in silico experiments. 
This can reduce the cost and time of carrying out joint action experiments and can 
be effectively used to test different human-machine interaction scenarios via the 
mirror game. 
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Supplementary Material 


The solution of the optimal control algorithm in each time interval [t/t,4+i] can 
be transformed into a boundary value problem by applying Pontryagin’s minimum 
principle Il25]l . We start by constructing the Hamiltonian as follows 


H{X,u,X) 




+ 


-{ax^ + py^ 


y ^ 

— y)y — aPx + u J 


where X = = [x,yY and A = [^ 1 ,^ 2 ]^. Using the minimum principle gives 

optimal open loop control 


u* =argminueRH{X*,u,X) = ^ j =-ri„PX2 
and optimal state equation 

^ ^ _|_ py*2 _ y'^y* _ ^ 2 ^* _ 1^2 

with initial condition X{tY = [x{tk),x{tic)Y and optimal costate equation 

X = -XxH = 


X2{2ax*y* + op-) 

A2(ax*2 + 3 py*^ -y)-Xi- dcj(y* - r^) 


with the terminal condition 

A ( 4 + 1 ) 


Op(x*(tk+i)-rp(tk+i)) \ 

0 J 


Let X denote the approximation of the optimal solution x*, then it is feasible to 
estimate the position error between the VP and the HP based on the collocation 
method as. 

\x —rp\ = \x —x + x — rp\<\x —x\ + \x — rp\ 

Notice that \x* — x\ is negligible due to the high approximation accuracy of nu¬ 
merical methods |[26]l . In particular, considering that normally the optimal so¬ 
lution X* is not available, the approximate solution x exactly corresponds to the 
position of the VP in the simulation. Thus, we mainly focus on the estimation 
of \x — rp\. For simplicity, we define x{t) = ao + ai{t — 4 ) -|-a 2 (f — 4 )^, Ai(t) = 
^ 1 (f - 4 ) + ^2 (f - 4 )^ and A 2 (t) = Co -h Cl (f - 4 ) -h C 2 (t - 4 )^, where a,-, bi and 
Ci, i G {0,1,2} are unknown constants and t G [ 4 , 4 + 1 ]. Substituting x{t), X\{t) and 
A 2 (t) into the above optimal state equation and costate equation at the boundary 
points yields the linear matrix equation 


AkXk = Bk 


(16) 
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where 


/ 1 

0 

0 

0 

0 

0 

0 

0 

0 

\ 

0 

1 

0 

0 

0 

0 

0 

0 

0 



BpT 

dpT^ 

-1 

-T 

-T^ 

0 

0 

0 


0 

0 

0 

0 

0 

0 

1 

T 

j2 


0 

0 

2 

0 

0 

0 


0 

0 


0 

0 

0 

0 

1 

0 

-(2ax{tk)y{tk) + ( 0 ^) 

0 

0 


0 

0 

0 

1 

0 

0 

-{ax{tkf + 3[5y{tkf - y) 

1 

0 


^p 

Tdp + Qa 

T{Tdp + 2ea) 

0 

0 

0 

0 

1 

2T 


\ 0 

0 

0 

0 

1 

2T 

0 

0 

0 

/ 


and 



( x{tk) \ 


/ ao \ 


y{tk) 


ai 


dpPp 


a2 


0 


bo 

Bk = 

-{ax{tkf + pyitkf - y)yitk) - (o^x{tk) 

, = 

b\ 


0 


b2 


-Ba{y{tk)-ra{tk)) 


Co 


Bpfp + Bara{tk+\) 


Cl 


V 0 J \C2 J 


Solving equation (fT^ determines the vector of unknown constants 

Xk=A^^Bk 


Thus, we obtain the approximate solution 

x{t) =x{tk)+y{tk){t-tk) + —{t-tk) , t G [tk,tk+i] 

where 

^ + {rp-x{tk) - Ty{tk))Bp] 

T^co^ o , 7 

— 'Hmi — 2 — —yT-\-2) 

■ [(ax(tkf + py(tkf - y)y(tk) + (0^x{tk)] 


and 

T^(Xp- 

^ = n'^iQpT + 0(j) + 2 t 7 „,(—^-h aT^x{tk)y{tk) + (xTx{tk)^ + 'i^Ty{tk)^ — yT+2) 
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Then we can compute 

\x{tk+\)-rp{tk+\)\= lim \x{t)-rp{tk+i)\ 

t^h+i 

= \x{tk) + Ty{tk) + ^T^-rp{tk+i)\ 

<T ^+ 

(17) 


where 

T^co^ 

^- V (xT^x{tk)y{tk) + (xTx{tk)^ + 3/3rj(4)^ — jT + 2 

and 


^ = 2{x(tk) + Ty{tk) - rp{tk+i)) - T^{y{tk)iax{tkf + I5y{tk)^ - 7 ) + ( 0 ^x{tk)) 

Since Xp, r^, S’, ^ and ^ are all bounded, it follows from inequality (fTTI) that 
the bound on the tracking error |jc(4+i) — fp(4+i)| converges to 0 as 0p ^ 1 and 
Tjm —>■ 0. Similarly, we can estimate the velocity error between the VP and the 
reference signal encoding the desired signature as follows 


\x{tk+i) - ra{tk+i)\ = lim \x{t)-ra{t)\ 

= \yitk)+ '^^T - ra{tk+l)\ 


<{i-ea 


2T'^\T{y{tk)-rc{tk+i))+2{rp{tk+i)-x{tk)-Ty{tk))\ 


, n 2T'^\ra{tk)-y{tk)\ , ,, 

“T C/(j I ^1 + 


M 


M 


(18) 


where 


^ =y{tk) - r(y{tk+i) - T[{ax{tkf + ^y{tkf - Y)y{tk) + 0)^x{tk)] 
According to inequality (fT^ . the bound of the velocity error goes to 0 if 6(y —^ 

ri,„ ^ 0 and ra (tk) =y( 4 )- 
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